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ABSTRACT 

This paper identifies some issues that may prevent the 
implementation of evaluation results in government agencies. A major reason 
for the lack of implementation in such agencies often is that evaluations are 
expected to be used both for information purposes and as accountability 
mechanisms. These two purposes do not always work together. When this is the 
case, the focus of the evaluation, the way it is conducted, and the manner in 
which the results are disseminated may not be conductive to program personnel 
implementing the results. There are other reasons why evaluations may not be 
implemented in government agencies, and this paper suggests some strategies 
for dealing with these problems, including the presentation of the results in 
varied settings and formats and, where, appropriate, the establishment of 
steering committees or project groups to ensure that the results are 
implemented appropriately. (Author) 
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Executive Summary 



This paper identifies some issues that may prevent the implementation of evaluation 
results in government agencies. A major reason for the lack of implementation in such agencies 
often is that evaluations are expected to be used both for information purposes and as 
accountability mechanisms. These two purposes do not always work together. When this is the 
case, the focus of the evaluation, the way it is conducted, and the manner in which the results are 
disseminated may not be conducive to program personnel implementing the results. There are 
other reasons why evaluations may not be implemented in government agencies, and this paper 
suggests some strategies for dealing with these problems, including the presentation of the results 
in varied settings and formats and, where appropriate, the establishment of steering committees 
or project groups to ensure that the results are implemented appropriately. 



The purpose of this paper is to make suggestions that might assist in the findings of 
evaluations being implemented. The lack of implementation is not a new problem. It has always 
been an issue. There are a number of reasons why nonimplementation of findings can occur, and 
this paper outlines some of these reasons and ways of dealing with them. 



Purpose of Evaluation 

Basically there are two main purposes for evaluations in most government agencies, such 
as NSF, the Department of Education, or most state agencies of education. Evaluations are for 

• accountability (summative), or 

• improvement (formative). 

Often evaluations are expected to do both, and this dual purpose can create problems in the focus 
of an evaluation, how it is carried out, how it is reported, and how it is implemented. 

It is important when doing an evaluation to establish the primary purpose at the outset. 
Although one purpose, such as accountability, does not rule out the other completely, it can make 
a considerable difference in how the evaluation is conducted and how results are communicated. 
Usually, if an evaluation is summative, it is more formal, and the findings are likely to be those 
required by stakeholders (e.g., legislators), with the focus being on how the program is delivering 
results. For example, achievement on standardized test scores is a typical requirement of 
summative evaluations in education, but these scores may not suggest in detail how to improve a 
program. If the evaluation is formative, the main purpose is to provide information for program 
and project managers to help them make decisions for improvement, and the focus tends to be on 
program processes (i.e., how effectively the program is being delivered, as opposed to 
outcomes). 
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Reasons for Nonimplementation of Findings 

There are a number of reasons why evaluations are not implemented, and they fall into 
four categories. 

* The evaluation is not considered relevant. 

* The evaluation is considered to be solely for accountability, and it is not necessary to 
implement the findings, e.g., standardized test scores are improving significantly, which 
does not suggest program changes. 

* The evaluation produces results decision makers do not like, often for political reasons, 
e.g., the program is performing better or worse than decision makers’ own personal 
opinions about the program. 

* A combination of two or more of these factors. 



Lack of Relevance 

One of the major reasons why evaluations are not implemented is because they are seen 
to be irrelevant. There are a number of reasons why this might be the case. 

Mismatch of Evaluators and Program Personnel 

There may be a mismatch of roles and styles between evaluators and program personnel, 
and this mismatch inhibits the ability of the program personnel to implement the findings. For 
example, the evaluators collect qualitative data that they find persuasive about the success of the 
program, but the top decision makers want "hard data" that the public might find more 
persuasive. Or the evaluators tend to be more directive than the program managers find tolerable. 

Timeliness 

The evaluation results are too late to be of any use to program personnel. Program 
personnel often have access to ongoing information about the effectiveness of their programs 
through meetings, telephone calls, the media, public interest groups, professional groups, etc., 
and when the evaluation results are published, the findings are "out of date." For example, many 
large-scale evaluations take a long time to conduct, and decision makers may not be willing to 
wait five years for the findings. When the findings do come, key decisions have already been 
taken. 

Inconclusive Results 

The findings are not conclusive enough to allow for implementation, or they are not 
rigorous enough to be convincing. In the case of evaluations that are conducted for the purpose 
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of accountability, the findings do not provide recommendations, or the recommendations are so 
broad that the program personnel have difficulty connecting them to their activities. Evaluators 
tend to be even-handed about findings, e.g., “on the one hand. ...” Many decision makers find 
this an evasion of responsibility. 

Asking the Wrong Questions 

The results are regarded as irrelevant because the evaluation asked questions that were of 
little interest to program personnel. The information required by legislators and stakeholders is 
often different from that required by program personnel, and it may be difficult for program 
personnel to use the information. Legislators often want “hard data” such as test scores or 
numbers that appear firm to them, and often evaluations do not produce these scores because of 
the difficulty and expense of doing so. 

Communication 

The evaluation results are not communicated in such a way as to be useful. 
Communicating the results is almost as important as ensuring that the evaluation has been 
conducted in a valid manner. The best conducted evaluations can gather dust if the information 
they provide is not presented in a way that suits the audience it is intended for. If the evaluation 
is intended for both summative and formative purposes, consideration should be given to 
presenting the information in different formats for different audiences. Providing lengthy 
evaluation reports filled with detail often does not provide the impact one is looking for. Face-to- 
face presentations carry more impact with busy decision makers. 

Other Factors 

The evaluations are not considered important because of factors such as funding or 
political issues. For example, some programs are funded for political reasons, and these may 
override the evaluation results. Programs for the disadvantaged are an example. It is unlikely 
such programs can be discontinued, regardless of the findings, because the political 
considerations are too important. Some problems might be mitigated if they are conducted by a 
unit also part of the organization. Internal evaluators should be aware of internal issues and 
realities and take them into account in the design of evaluations, in their briefing of external 
evaluators, and in communicating results. 



Accountability vs. Information 

If the reason evaluations are not being implemented is that they are considered an 
accountability mechanism rather than information that needs to be acted on, then no further 
action need be taken. For example, program managers often see reporting data to legislators as 
part of the routine of doing business, which requires no further action. However, it may be that 
evaluations regarded in this way by program personnel could be used for revising the program if 
the data are appropriate. 
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If evaluations are meant to be for both accountability and improving the program, there 
may be conflict in attempting to meet both requirements. If evaluations are accountability 
mechanisms, then it may be desirable for the evaluator to be arm’s length from program 
personnel. This distance precludes a more collaborative approach, which is important if the 
evaluation is expected to monitor a program's implementation, test for progress in the 
achievement of results, and identity potential problems. 



Evaluators Produce Results Decision Makers Do Not Like 



This situation can arise if the evaluation is mainly an accountability mechanism, but it 
should arise less often if the evaluation is regarded as an information-gathering tool. Decision 
makers may already have made up their minds regarding the success of the program, or the 
politics may be such that findings make little difference one way or the other. Internal evaluators 
should be able to anticipate such possibilities. Even if the evaluation is meant to be purely an 
accountability mechanism, there should be no surprises for program personnel. If evidence arises 
that points to negative results, these should be discussed with program personnel before a final 
report is produced. Some of these problems involve the role of internal evaluators. 



Internal Evaluators 

Evaluators internal to the agency have some advantages. They are part of the organization 
and understand many aspects of it, including the policies, processes, and personnel. This position 
allows them to design evaluations that suit the organizational situation and develop working 
relationships with program personnel. For example, they should know that a particular division 
head is highly suspicious of evaluation in general and should plan accordingly. 

The role of evaluator within an organization is somewhat different from that of external 
evaluator. An external evaluator is not subject to the hierarchy of the agency to the same degree, 
although dependent on obtaining contracts from it, perhaps. Internal evaluators must be 
concerned with internal issues, including the process of the delivery of services and providing 
agency management with appropriate information for decision making. Internal evaluators are in 
a better position to consider factors that external evaluators may not be privy to or may not 
regard as important, such as political and funding issues specific to the agency. 

At the same time, evaluations that take place within an organization are open to questions 
of credibility to outside audiences, since the evaluators may be seen as too much a part of the 
organization. The problem of credibility is partly compensated by the advantages an internal 
evaluator has in developing working relationships with program personnel, providing relevant 
and timely information to program managers, and taking into account the various aspects of the 
organization that an outsider would not know. The internal evaluator may provide information 
that is more credible to those inside the agency. 

Having an internal evaluation unit that outsources most evaluations could provide a blend 
of both worlds. The internal evaluation unit has the benefit of understanding the nature of the 
organization, knowledge of its policies and processes and program personnel. Conversely, by 
using external evaluators, credibility can be maintained with external audiences. However, such 
a situation can work the other way if knowledge of the organization and its practices and 
program personnel are not utilized to advantage. The evaluation can be seen as not credible to 
outside arid inside audiences alike. 

In general, there are two purposes, summative and formative, and two positions for 
evaluators, internal or external to the agency. It seems natural enough that summative evaluation 
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has more credibility for outsiders when conducted by external evaluators less beholden to the 
organization, and that internal evaluators would have knowledge that leads to credibility with 
those inside the agency, especially for program improvement. Contracting out evaluations 
presents the challenge of doing all these tasks simultaneously. 



Conduct of Evaluations 

Who Is the Audience? 

In conducting an evaluation it is desirable at the outset to be clear about the primary 
audience, because audience will determine the focus of evaluation, the way it is conducted, and 
the manner in which the findings are communicated. If the primary audience consists of program 
personnel, either at the agency or project level, then usually what is required is monitoring the 
program, periodic testing to check that it is achieving what it is meant to achieve, and identifying 
potential problems. In this instance a final written report may or may not be required, but an 
important task would be providing feedback to program personnel. 

Evaluation for external audiences has a different focus. The primary audiences are 
stakeholders and legislators who usually require reports providing information on the general 
effects and achievements of a program. In such a case, a final written report is almost certainly 
required. Even here, however, it may be appropriate to provide informal feedback to these 
outsider audiences, such as legislative staff, during the course of the evaluation. 

Understanding the Audiences and Their Requirements 

In conducting an evaluation it is important to know the audiences and understand their 
requirements. The requirements of program managers at a program and project level will be 
different from those of outside stakeholders. Program and project managers usually require 
information for decision making, which makes timeliness crucial. Being able to provide 
information on an ad hoc basis may not be possible in certain types of evaluations, especially 
with large program evaluations that have their primary focus on outcomes. 

Developing Approach Strategies 

Knowing the primary audiences for the evaluation and understanding their requirements 
should enable the evaluator or team of evaluators to determine the best approach in conducting 
the evaluation. Thus, some contact with these audiences is usually required to ascertain what it is 
they consider critical for the evaluation. Stakeholder information needs are not the only 
considerations in designing an evaluation, but they are an important aspect. It does little good to 
collect information, such as the opinion of project personnel, that the audiences will consider not 
useful or inherently biased. 

Operating Principles 
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Regardless of the type of evaluation, the following principles may assist in results being 
implemented. 

Involvement. Involve program personnel. Program personnel should be involved to 
provide knowledge and also so they feel part of the evaluation. They are much more likely to 
implement the findings. 

Publicity. Make findings known to higher authorities. It matters not only that one knows 
what to do but also that one’s superiors also know that one knows. Expectations, known and 
unknown, are a big part of implementing findings. 

Trust. Do not surprise, embarrass, or attack program personnel at the agency or project 
level. No one wants to be embarrassed, especially about one’s work. Evaluators should apply the 
same consideration to clients and audiences as teachers do to students. 

Time. Allow findings to seep into comprehension over time. One of the most overlooked 
aspects of evaluation is that people need time to assimilate new information, especially if it is - 
negative. The first reaction to bad news is emotional, and reflecting on the findings over time 
allows a reasoned acceptance. 

Early Consultation 

Early meetings with program personnel in the agency about the evaluation — its nature, 
scope, anticipated outcomes, and what will be done with results — are very important. Ideas 
should be solicited from the program personnel and their cooperation enlisted. If possible, 
project personnel in the field should be consulted too, since often the agency and field views of 
what is happening are different. Since most agencies are seriously underfunded for travel, getting 
this information is not always an easy task. In the case of formative evaluations especially, it is 
important to find out what information the agency personnel require and what questions they 
want answered. 

Feedback 

In any evaluation, progress reports on an informal basis should be given at strategic 
points in the evaluation, preferably face-to-face. There should be no surprises for program 
personnel at the end of the evaluation. If one has negative findings, the agency personnel should 
be given an opportunity to respond to these, to raise questions, and to blow off steam before the 
findings are made public. There is a simple courtesy of not embarrassing program personnel. It 
does not help evaluation in the long run or build an evaluation culture inside the organization. 
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Dissemination of Results 



There are many ways of disseminating the results of an evaluation, and the method 
chosen should be one that best suits the audience. When considering how to communicate the 
results, the following questions should be asked: 

Is the information relevant to the user's needs? (Not everyone needs to know everything.) 

Is the information applicable, i.e., can the user apply the information to the current 
situation? (General findings often are too vaguely stated for program personnel to do anything 
with them.) 

Is the information understandable to the user? (Means and standard deviations don't mean 
much to lay audiences. Provide specific concrete examples of good, average, and bad projects, if 
that happens to be the focus of the evaluation. Instead of presenting means, try assigning 
well-known evaluative categories, such as letter grades, e. g., A, B, C.) 

Is the information timely? (Few people will wait five years for evaluative information. 
The world changes too fast, especially in the crisis management of government agencies. 
Evaluators should produce evaluation reports once a year, if at all possible. It reminds audiences 
that the evaluation is ongoing and prepares them for possibly negative results that will be 
forthcoming.) 

In answering these questions, it should be possible to determine formats for 
communicating results. For example, verbal presentations are better than written reports for 
communicating findings in a timely manner. They also provide an opportunity for 
users to ask questions if there are aspects of the findings that they do not understand or like. 

Verbal Presentations 

Verbal presentations are useful for communicating results. In some instances the 
audience may not be familiar with the technical aspects of the evaluation, and a face-to-face 
setting allows an evaluator to report aspects of the evaluation in a language that is accessible to 
all. Pictorial representations in such settings allow audiences to ask questions about aspects that 
may be transparent to evaluators but not to program personnel or lay audiences. 

Even when a formal written report is required, consideration should be given to 
presenting the information verbally. Presentations can be made at informal meetings with 
program personnel or seminars to which major stakeholders are invited. The composition of the 
seminars would be determined by the evaluation. For example, if the evaluation is for 
accountability, outside stakeholders and legislators would be audiences. 



In the case of evaluations for improvement, seminars provide opportunities for program 
personnel to identify aspects of the evaluation of most interest to them and to provide feedback 
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to evaluators on issues in which program personnel are most interested. Many disputes can be 
resolved at this stage. 

If the findings of an evaluation are made public, an informal preview of results should be 
presented to program personnel prior to public airing so that there are no surprises for them. It is 
easy for evaluators to underestimate the emotional impact of findings or even certain words. One 
way of ensuring that program personnel are not embarrassed is to allow them to preview the 
findings. The change of a few words here or there can work wonders for cooperation. 

Where appropriate, formal invitational seminars could be given that include significant 
outsiders. 



Written Presentations 

Most evaluations require written reports. Again, the audiences should be kept in mind. 

The report should be written with regard to particular audiences and consideration given to their 
information needs, their level of understanding, and the usefulness of the information to them.^ 
Agencies such as the Office of Technology Assessment have found the clarity and style of 
reports to be critical. 

Executive summaries are useful. Managers are busy, and either don't have the time to 
wade through data or do not have the technical expertise to interpret it. A copy of the report 
and/or an executive summary should be given to the head of the agency so that he/she is aware of 
what work has been done and what actions need be taken. Informing superiors is important for 
establishing expectations. Care should be taken to follow the appropriate chain of command in 
providing these reports, however. 

The results of evaluations should be discussed with relevant program personnel, division 
heads, and the head of the evaluation unit. The staff involved would be determined by the scope 
of the evaluation. If the evaluation is of a project that is a small part of an overall program, the 
involvement of more senior staff may be unnecessary. As a rule of thumb, if change is required, 
then anyone affected by the change should be briefed on the evaluation and its recommendations. 

Publishing the Evaluation 

Ordinarily, the evaluation should be published by whoever is authorized to do so, the 
agency, the external evaluator, etc. Publishing means the evaluation is made available to 
whoever wants it. If nothing else, distribution will create expectations that something will be 
done to implement the findings. Sometimes evaluations are done under confidential 
circumstances, but this should not ordinarily be the case. Distribution procedures should be 
worked out in the original understandings and contracts, not negotiated after the findings are in. 

Implementation Reports 
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It may be appropriate to indicate activities that could be undertaken to implement the 
findings, although often the evaluator will not have the expertise to know what to do. Knowing 
how to evaluate and how to improve a program are not the same thing. Nonetheless, it is helpful 
if program personnel are able to relate their activities to the findings. Such information could be 
given as part of the evaluation report or as a separate implementation report. 



Implementation of Results 

In some instances it may not be necessary or appropriate to implement findings or 
recommendations, but the decision should be taken by program managers, division heads, and 
agency heads. Not all evaluations need be implemented. The findings may simply say that the 
program was an overwhelming success with no discernible recommendations for improvement. 
Or it may be that findings have been communicated to program managers who have taken action 
already. When it is appropriate to implement the findings, the following suggestions may be 
useful. 



Implementation Committees 

If the evaluation is large in scope with a number of possible actions arising from the 
findings, consideration should be given to setting up an Implementation Committee to oversee 
implementation. Such a process is sometimes essential if action is required across divisional 
boundaries or is outside the jurisdiction of the group making recommendations. 

Composition of Implementation Committees 

An Implementation Committee should include major stakeholders, such as agency and 
program personnel who are to implement the findings. For example, such a committee for a 
minority education program might include agency program personnel, representatives from the 
office of the agency head, project personnel, representatives from the minority communities 
affected, and evaluators who have an understanding of the evaluation. 

Role of Implementation Committees 

The Implementation Committee would delineate what action is required to implement 
findings of the evaluation and who is responsible for implementing the actions, plus the time line 
in which action is to be completed. The committee would function as an oversight committee. 

Frequency of Meetings 

The complexity of actions required to implement the findings would determine the 
frequency of meetings. The benefit of an Implementation Committee is that everyone involved 
knows what is required, and there is an expectation that action will be undertaken by certain 
people within a certain time. Things won't drift into inaction. 
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Project Groups 



Where the implementation affects only one area of the agency, a project team may be all 
that is required. Again it should be composed of stakeholders and those expected to implement 
changes. It should be clear what action is required, who is responsible, and the time frame in 
which action should be undertaken. 



Review 

Assessing how the findings and recommendations from an evaluation were implemented 
is an important part of the process. An implementation review also provides evaluation personnel 
with feedback on how they should present their information in the future and what 
recommendations are appropriate. The review could be done by the evaluation office, by the 
Implementation Committee if one exists, or by an outside group or the agency head's office. 

Follow-up Surveys 

Review might be based on a follow-up survey. Such a survey should probably be 
undertaken about twelve months after the evaluation has been completed. The survey would 
focus on discovering whether the recommendations had been implemented, how they had been 
implemented, and what barriers prevented implementation if they were not. 

Follow-up Meetings 

An alternative to a survey would be a follow-up meeting that would focus on discovering 
whether the findings and recommendations of the evaluation were implemented, how 
recommendations were implemented, the barriers to implementation, and ideas for evaluation 
improvement. 



Summary 

These suggestions for implementing the findings and recommendations from evaluations 
within organizations are based on several considerations: involving program personnel in the 
evaluation, understanding the information needs of primary audiences, making findings known 
to higher authorities and major stakeholders, establishing bonds of trust between evaluators and 
program personnel, presenting results in an understandable manner, and allowing enough time so 
everyone can accommodate the findings mentally. No doubt there are many other strategies as 
well. 
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